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REGION-BASED AUTO GAIN CONTROL AND AUTO EXPOSURE 
CONTROL METHOD AND APPARATUS 

FIELD OF THE INVENTION 

[0001] The invention relates generally to automatic adjustment of exposure 
and gain, and in particular to a highly flexible region-based approach to automatic 
adjustment of exposure and gain for image sensors. 

BACKGROUND OF THE INVENTION 

[0002] CMOS imagers are low cost imaging devices. A fully compatible 
CMOS sensor technology enabling a higher level of integration of an image array with 
associated processing circuits would be beneficial to many digital applications such as, for 
example, in cameras, scanners, machine vision systems, vehicle navigation systems, video 
telephones, computer input devices, surveillance systems, auto focus systems, star trackers, 
motion detection systems, image stabilization systems and data compression systems for 
high-definition television. 

[0003] CMOS imagers have a low voltage operation and low power 
consumption; CMOS imagers are compatible with integrated on-chip electronics (control 
logic and timing, image processing, and signal conditioning such as A/D conversion); 
CMOS imagers allow random access to the image data; and CMOS imagers have lower 
fabrication costs as compared with, for example, the conventional CCD since standard 
CMOS processing techniques can be used. Additionally, low power consumption is 
achieved for CMOS imagers because only one row of pixels at a time needs to be active 
during the readout and there is no charge transfer (and associated switching) from pixel to 
pixel during image acquisition. On-chip integration of electronics is particularly 
advantageous because of the potential to perform many signal conditioning functions in the 
digital domain (versus analog signal processing) as well as to achieve a reduction in system 
size and cost. 
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[0004] In order to maintain the quality and brightness of an image at an 
optimal level, the exposure and gain settings have to be continually adjusted for varying 
light conditions. Exposure is the duration for which the pixel sensor is capturing photons 
and accumulating induced electrons. Gain is the amoimt of analog amplification or 

attenuation that a pixel sensor signal undergoes. Amplification is where the gain is greater 
than one and attenuation is where the gain is less than one. 

[0005] By varying the exposure and the gain of a pixel sensor, optimal 
images can be obtained from a sensor. For example, for the bright light conditions of a 
beach on a simny day, the exposure would be set to a minimum and the gain to less than or 
equal to one. Similarly, if the image desired to be captured is a polar bear in a snow storm, 
the exposure would be set to a minimum and the gain to less than or equal to one. For 
dark conditions such as when trying to capture an image of a deer at night, the exposure 
would be set to a maximum and the gain to greater than or equal to one. Automatic 
exposure and gain control algorithms, however, carry the risk of oscillations. If the desired 
exposure and gain and the actual exposure and gain do not converge, then oscillations 
result, which adversely impact the captured image. 

BRIEF SUMA4ARY OF THE INVENTION 

[0006] The apparatus and method of the present invention performs 
automatic exposure and gain control while minimizing oscillations as well as providing a 
good response time, for example, a lag time or a settling time of about one fi-ame. The 
automatic exposure and gain controls are performed not only on the image as a whole but 
on a weighted region of interest. If the contrast in the image exceeds the dynamic range of 
the sensor array, then the image in the region of interest will improve at the expense of the 
remainder of the image. A region of interest is a selected subset of tiles upon which 
automatic exposure and gain control will be based. The tiles are defined by a grid system 
having grid coordinates, which are programmable. 

[0007] Image sensors have to receive feedback with regular updates of 
exposure and gain settings based on ever changing light conditions. Concentrating on a 
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region of interest ensures that the object of interest (the target of the photo) is correctly 
exposed. Therefore, if the object of interest becomes backlit or shaded then the region of 
interest can be defined such that the object of interest is correctly exposed even at the 
expense of the remainder of the image. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0008] Other features and advantages of the present invention will become 
apparent when the following description is read in conjunction with the accompanying 
drawings, in which: 

[0009] FIG. I shows a selection of tiles based on an object of interest in a 
region of interest; 

[0010] FIG. 2 is a block diagram of an auto-control feedback loop; 

[0011] FIG. 3 is an example of an array of image tiles indicating weights 
assigned to tiles of an image; 

[0012] FIG. 4 is an example of a histogram built in accordance v^th the 
present invention; 

[0013] FIG. 5 is an example of a histogram used to accumulate a 
score/count at a target percentile in accordance with the present invention; 

[0014] FIG. 6A - E are flowcharts illustrating the method of performing 
automatic exposure and gain control in accordance with the present invention; 

[0015] FIG. 7A is a block diagram of the automatic exposure and gain 
control circuit of the present invention; 

[0016] FIG. 7B is a block diagram of a portion of the automatic exposure 
and gain control circuit of FIG. 7A; 



1636587 v1;Z2SR01I.OOC 



3 



Docket No.: M4065.0975/P975 
[0017] FIG. 8 is a block diagram of an exemplary digital camera system 
having a dark current and defective pixel compensation circuit of the present invention; and 

[0018] FIG. 9 is a block diagram of a computer system utilizing an imager 
having a automatic exposure and gain control circuit of the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

[0019] FIG. 1 shows a selection of tiles based on an object of interest 110 in 
a region of interest 115 (shaded tiles). Initially, the tiles 105 are defined by dividing the 
image field into tiles of equal size as illustrated in the tiles of the image field on the left of 
FIG. 1. Once a region of interest 115 has been identified based on the selection of an 
object of interest 110, the size of the tiles can be adjusted by adjusting the grid coordinates 
120 of the tiles as illustrated in the tiles of the image field on the right of FIG. 1. The tiles 
of the region of interest 115 (shaded tiles) have been made smaller and the tiles outside of 
the region of interest 115 have been made larger. 

[0020] FIG. 2 is a block diagram of an auto-control feedback loop 200. The 
sensor array 205 has pixels, which accumulate charge based on an exposure to light. 
Analog exposure and gain registers 210 control the exposure and gain of the sensor array 
205. An image stream 215 from the sensor array 205 is applied to the region-based 
automatic exposure and gain control circuit 220. The analog exposure and gain registers 
210 are updated based on feedback 225 from the region-based automatic exposure and 
gain control circuit 220. The region-based automatic exposure and gain control circuit 
220 receives an input 230 fi-om a target application that defines the region of interest based 
on identification of a target of interest. 

[0021 ] Exposure and gain values are computed on the basis of pixels in a 
selected region of interest that carry weights with respect to one another, A histogram is 
constructed based on the pixels located in the region of interest, and exposure and gain 
values are computed based on the histogram. Not all pixels in the region of interest are 
required to compute the exposure and gain. Therefore, the pixels in the region of interest 
are sampled. For example, there are 640 x 480 pixels in a Video Graphic Array (VGA) 
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image. Instead of using all of the pixels, a sampling is made of one pixel in every nth row 
and one pixel in every mth column, where n and m may be equal. For example, a 
sampling of one pixel in every 16th row and one pixel in every 16th column is taken 
yielding a sampling of 40 x 30 pixels out of the 640 x 480 total pixels. The dynamic range 
of each pixel is defined by the number of bits for each pixel signal value. For example, a 
dynamic range of 0 - 1023 would require 10 bits per pixel. 

[0022] FIG. 3 is an example of an array of image tiles 300 indicating weights 
assigned to the tiles 305 of an image. The higher the weight the more interest there is in a 
particular tile. The image is sub-sampled based on the tile weights and the sample 
described above. For example, if the weight of the tile is zero then no pixels in that tile will 
be included in the sub-sample. If the tile has a weight of sixteen, however, then 16 pixels 
from that tile are included in the sub-sample. Similarly, for other tile weights up to the nth 
sample in a row and the mth sample in a column, the pixels of the sample are sub -sampled. 
That is, the sub-sample will contain no more samples than the sampling defined by n and m 
and discussed above. The sub-sampling based on tile weight allows a concentration or 
focusing on the object of interest contained in the region of interest. 

[0023] A histogram representing bin numbers vs. pixel count is constructed 
for every image. A bin is an aggregation of frequencies. In order to decrease the 
settling/lag time, the count need not be based on the entire dynamic range but is instead 
based, for example, on the six most significant bits of a ten bit readout register. The look 
of the histogram is different based on the bin width. The larger the bin width the coarser 
the histogram. The narrower the bin width, the closer the histogram will approximate a 
frequency distribution. It is not imreasonable to set the bin width in the neighborhood of 
sixteen. Further, the size of each bin need not be the maximum possible of 40 x 30 but 
need only be 10% or 120. This is based on using a 90th percentile to determine the bin 
number that is required to accumulate a count of 120. 

[0024] FIG. 4 is an example of an histogram built in accordance with the 
present invention. The histogram of FIG. 4 uses the six most significant bits of the 
dynamic range of each pixel signal value in the sub-sample. Bin number 63 has the count 
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Bin number 0 has the count of the 



[0025] FIG. 5 is an example of a histogram used to accumulate a 
score/count at a target percentile. Based on using a 90th percentile and staring at bin 
number 63, the counts from each bin are accumulated until a count of 120 has been 
reached. 

[0026] There are two main procedures performed by the invention - 
automatic exposure control and automatic gain control. There is an overriding procedure 
that controls which of the procedures is static and which is operational/activated at any 
given time. Automatic exposure control takes precedence over automatic gain control. 
When automatic exposure control is activated, automatic gain control is set at one (unity) 
implying no gain or attenuation. If light conditions are very bright, then the exposure is 
automatically reduced accordingly. If light conditions are very dark then the exposure is 
automatically increased accordingly. It is only when the exposure reaches a limit 
(minimum or maximum) that automatic gain control is activated. The gain values range 

from — to 15 — with a preferred gain value in the range of — to — . 
16 16 ^ ^ ^16 16 

[0027] FIG. 6A - E are flowcharts illustrating the method of performing 
automatic exposure and gain control in accordance with an embodiment of the present 
invention. FIG. 6B is a continuation of FIG. 6A. FIG 6C is a continuation of FIG. 6B. 
FIG 6D illustrates the automatic gain control method and FIG. 6E is a continuation of Fig. 
6D. 

[0028] The method begins at step 605 by initializing the analog exposure 
and gain control registers. The gain is set to one and the automatic exposure control 
method is activated at step 610. The pixels of the image are sampled at every nth row and 
every mth column at step 615, where n and m may be equal. The array of weighted tiles 
for the image is retrieved at step 620 to determine the nimiber of pixels from each tile to 
include in the sub-sample. The histogram is constructed at step 625 using the x most 
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significant bits of the dynamic range of pixel signal values. For example, using the six most 
significant bits means that the 4 least significant bits are unused. This is equivalent to 
division by 16. 

[0029] Referring now to FIG. 6B, the bin number to accumulate 10% of the 
pixels is determined at step 630. For example, using 90th percentile, a coimt of 120 pixels 
would be required. Using 95th percentile, a count of 60 pixels would be required. The 
bin number determined in step 630 is compared to bin number 60 at step 635. If the bin 
number is greater than bin number 60, then the exposure is reduced by multiplying the 
current exposure by 0.5 at step 640. The exposure is tested at step 645 and if the exposure 
has not reached a minimum limit at this point, then the bin nvunber is recomputed using a 
selected percentile at step 630. If, at step 645, the exposure has reached a minimum, the 
automatic exposure control method is deactivated (made static/parked) and the automatic 
gain control method is activated at step 650. The automatic gain control method is shown 
in FIGs. 6D and 6E. 

[0030] If, at step 635, the bin number is less than or equal to 60, then the 
bin number is compared to bin number 6 at step 655. If the bin number is less than 6, 
then the exposure is increased by multiplying the current exposure by 16 at step 660. The 
exposure is tested at step 665 and if the exposure has not reached a maximum limit at this 
point, then the bin number is recomputed at step 630 using a selected percentile. If, at 
step 665, the exposure has reached a maximum, then the automatic exposure control 
method is deactivated (made static/parked) and the automatic gain control method is 
activated at step 670. 

[0031] Referring now to FIG. 6C, the bin number determined in step 630 is 
compared to bin numbers 7 and 16 at step 675. If, at step 675, the bin number is greater 
than bin number 7 and less than bin number 16, then at step 680 the exposure is increased 
by multiplying the current exposure by 2. If, at step 675, the bin number is greater than or 
equal to 16, then the exposure is adjusted by multiplying the current exposure by the 
threshold (or target bin number, for example bin number 50) and dividing by the current 
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The method continues at step 665 (described above with respect to FIG. 



[0032] Referring to FIG. 6D, the bin number to accumulate 10% of the 
pixels is determined at step 632. For example, using 90th percentile, a covmt of 120 pixels 
would be required. Using 95th percentile, a count of 60 pixels would be required. The 
bin nxmiber determined in step 632 is compared to bin number 60 at step 634. If the bin 
number is greater than bin number 60, then the gain is reduced by multiplying the current 
gain by 0.5 at step 636. The gain is tested at step 638 and, if the gain has not reached a 
minimum limit at this point, the bin number is recomputed at step 632 using a selected 
percentile. If the gain has reached a minimum (step 638), then the automatic gain control 
method is deactivated (made static/parked) and process is stopped at step 648. The 
automatic exposure control method is shown in FIGs. 6B and 6C. 

[0033] If, at step 634, the bin number is less than or equal to 60, then the 
bin number is compared to bin number 6 at step 642. If the bin number is less than 6 then 
the exposure is increased by multiplying the current exposure by 16 at step 644. The gain 
is tested at step 646, and if the gain has not reached a maximum limit at this point, the bin 
number is recomputed at step 632 using a selected percentile. If the gain has reached a 
maximum then the automatic gain control method is deactivated (made static/parked) and 
the process is stopped at step 648. The process is stopped at step 648 because the 
automatic exposure control method has already been executed. The automatic exposure 
control method is shown in FIGs. 6B and 6C. 

[0034] Referring now to FIG. 6E, the bin number determined in step 632 is 
compared to bin numbers 7 and 16 at step 652. If the bin number is greater than bin 
number 7 and less than bin number 16, then at step 654 the gain is increased by 
multiplying the current exposure by 2 at step 680. If the bin number is greater than or 
equal to 16, then the gain is adjusted at step 656 by multiplying the current gain by the 
threshold (or target bin number, for example bin number 50) and dividing by the current 
bin number. The method continues at step 646 (described above with respect to FIG. 
6D). 
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[0035] FIG. 7A is a block diagram of the automatic exposure and gain 
control circuit of the present invention. A sensor array 705 is an array of pixels used to 
capture an image. The sensor array 705 is coupled to a timing and control circuit 710 
having analog exposure and gain control registers 715. The sensor array 705 receives an 
input 740 from the timing and control circuit 710 indicating an exposiu-e control value, 
which is applied to the transistor gates of the pixels of the array to control how long the 
pixel accumulates charge. The exposure control value is applied to all pixels of the array if 
the sensor array uses a global shutter and to particular rows of the array if the sensor uses a 
rolling shutter. The analog-to-digital (A/D) converter 720 receives an input 745 from the 
timing and control circuit 710 indicating a gain control value, which is, for example, in the 

range of ~ 16 ' A/D converter 720 applies the gain control value to the analog 

pixel signal values 730 it receives from sensor array 705 and outputs digital pixel signal 
values to the automatic exposure control and automatic gain control circuit 725. One 
implementation of the automatic exposure control and automatic gain control circuit uses a 
Field Programmable Gate Array (FPGA). The automatic exposure control and automatic 
gain control circuit calculates exposure control and gain control adjustment values 750, 
which are applied to adjust values in the analog exposure control and gain control registers 
715. 

[0036] FIG. 7B is a block diagram of the automatic exposure and gain 
control circuit 725 of FIG. 7 A. The digital signal values are input to the automatic 
exposure and gain control circuit 725 via a data stream 785 via a video in module 770. 
The data stream 790 of digital signal values is forwarded to the pixel coordinate grabber 
765 and the video out module 760. The video out module 760 gets the data stream of 
digital signal values out of the automatic exposure and gain control circuit 725. That is, 
while the data stream is used to provide an adjustment to the analog exposure and gain 
control registers, that adjustment has approximately a one frame lag or settling time while 
the automatic exposure control and automatic gain control circvdt 725 is active. The data 
stream for the current frame is thus output directly to the video out module 760, the data 
stream for the current frame having been captured using the automatic exposure control 
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and automatic gain control adjustments calculated during the last frame. The data stream 
may be progressive or interlaced depending on the scan mode. An interlaced data stream 
starts with, for example, line 1 and then takes line 3 and continues in this manner until data 
has been obtained for the image and then goes back to take data for the even lines in this 
way collecting data for the entire image. A progressive data stream starts from the top at 
the left end and scans to the right and then starts on the next line and performs the same 
operation for that line imtil all data for an image has been collected. 

[0037] The pixel coordinate grabber 765 translates the data stream 790 of 
digital signal values to x-y coordinates 800, 805 using the horizontal and vertical sync 
signals. The x-y coordinates 800, 805 as well as a data stream 795 of digital signal values is 
forwarded to histogram generator 780. The histogram generator 780 generates a 
histogram of, for example, 63 bins, each bin being 120 deep. The histogram is used in 
conjtmction with the AEC/AGC module 775 to determine an adjustment for each the 
exposxu-e and the gain to be applied to analog exposure and gain control registers, which in 
turn apply the values to the transistors of pixels of the sensor array to control the duration 
of charge accumulation. The AEC/AGC control module 775 controls whether exposure 
or gain is adjusted based on exposure adjustment taking priority over gain adjustment. The 
global control registers 755 define the coordinates of the tiles of the image and assign 
weights to the tiles for sub-sampling as described above. The grid coordinates of the tiles 
of the image are programmable. The global control registers 755 interface with the 
AEC/AGC module 775 supplying the grid coordinates of the tiles of the image and the 
weights of the tiles of the image via an input 820. The AEC/AGC module 775 supplies 
feedback to the global control registers to adjust the size of the tiles and the weight via an 
output 815. The global control registers 755 also supply the grid coordinates of the tiles 
and the weights of the tiles to the histogram generator 780. The output of the AEC/AGC 
module 775 goes to the analog exposure and gain control registers of the timing and 
control circuit of the sensor array to provide an adjustment to the analog exposure and gain 
control registers. 
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[0038] FIG. 8 is an exemplary embodiment of a portion of a digital camera 
system 830, which employs an image processor 870 and has an automatic exposure control 
and automatic gain control circuit 875 in accordance with the present invention. The 
illustrated components could be integrated together in one integrated circuit or could be 
implemented with discrete components, A row decoder 835 and column decoder 840 are 
coupled to a pixel array 845 and are used to select a pixel in the pixel array 845. Each pixel 
outputs a pixel reset signal V„t and a pixel image signal V^ig. An array controller 850 is 
coupled to the row decoder 835 and column decoder 840 and determines which row and 
column are activated to produce the V„t and V^^^ signals. The sample/hold circuit 855 
sequentially receives the pixel signals from the column lines through the column decoder 
840 selection circuit. The sample/hold circuit 855 provides the V„t and V^jg pixel signals to 
subtractor 860, which subtracts the signals. The analog-to-digital (A/D) converter 865 
accepts the signal from subtractor 860 and outputs digital signals representing the 
subtracted V„t and V^jg pixel signals to the image processor 870. The image processor 870 
is also coupled to the automatic exposure control and automatic gain control circuit 875 
and output serializer 880. The automatic exposure control and automatic gain control 
circuit 875 is further coupled to analog exposure control and gain control registers 895, 
which are, in turn, coupled to a timing and control circmt 890 of the sensor array 845. 
The automatic exposure control and automatic gain control circuit 875 provides an 
adjustment to the analog exposure control and gain control registers via an output 897. 
The timing and control circuit 890 controls the gain with a gain control signal 891 applied 
to the analog- to-digital converter 865. The timing and control circviit 890 controls the 
exposure with an exposure control signal 893 applied to the pixel array 845. 

[0039] The exemplary image signal processing methods and apparatus 
described above may be implemented in software, hardware, firmware, application specific 
integrated circuits (ASICs), field programmable gate arrays (FPGAs) or any combinations 
of the above or their equivalent. 

[0040] A typical processor based system 900 that includes a CMOS imager 
device 910 according to the present invention is illustrated generally in FIG. 9. A 
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processor based system 900 is exemplary of a system having digital circuits that could 
include CMOS imager device 910 the details of which are described above with reference 
to FIGS. 1 through 8. 

[0041] A processor system 900, such as a computer system, for example, 
generally comprises a central processing imit (CPU) 944 that communicates with an 
input/output (I/O) device 946 over a bus 952. The CMOS imager 910 also 
commimicates with the system over bus 952. The computer system 900 also includes 
random access memory (RAM) 948, and, in the case of a computer system may include 
peripheral devices such as a floppy disk drive 954 and a compact disk (CD) ROM drive 956 
which also communicate with CPU 944 over the bus 952. Software could be stored on a 
floppy disk drive 954 or an a CD-ROM drive 956 for execution on, for example, image 
processor 870 (FIG. 8). 

[0042] It should again be noted that although the invention has been 
described with specific reference to CMOS imaging devices, the invention has broader 
applicability and may be used in any imaging apparatus. The above description and 
drawings illustrate preferred embodiments of the present invention. It is not intended that 
the present invention be limited to the illustrated embodiments. Any modification of the 
present invention that comes within the spirit and scope of the following claims should be 
considered part of the present invention. 
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